{
  "cells": [
    {
      "cell_type": "markdown",
      "id": "84cca40c",
      "metadata": {},
      "source": [
        "# Predicting which records match\n",
        "\n",
        "<a target=\"_blank\" href=\"https://colab.research.google.com/github/moj-analytical-services/splink/blob/master/docs/demos/tutorials/05_Predicting_results.ipynb\">\n",
        "  <img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/>\n",
        "</a>\n",
        "\n",
        "In the previous tutorial, we built and estimated a linkage model.\n",
        "\n",
        "In this tutorial, we will load the estimated model and use it to make predictions of which pairwise record comparisons match.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "id": "9a445f52",
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-06-22T08:08:04.047347Z",
          "iopub.status.busy": "2024-06-22T08:08:04.047019Z",
          "iopub.status.idle": "2024-06-22T08:08:04.053987Z",
          "shell.execute_reply": "2024-06-22T08:08:04.053286Z"
        },
        "tags": [
          "hide_input"
        ]
      },
      "outputs": [],
      "source": [
        "# Uncomment and run this cell if you're running in Google Colab.\n",
        "# !pip install splink"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "id": "48f57034",
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-06-22T08:08:04.057741Z",
          "iopub.status.busy": "2024-06-22T08:08:04.057465Z",
          "iopub.status.idle": "2024-06-22T08:08:06.111293Z",
          "shell.execute_reply": "2024-06-22T08:08:06.110501Z"
        },
        "tags": []
      },
      "outputs": [],
      "source": [
        "from splink import Linker, DuckDBAPI, splink_datasets\n",
        "\n",
        "import pandas as pd\n",
        "\n",
        "pd.options.display.max_columns = 1000\n",
        "\n",
        "db_api = DuckDBAPI()\n",
        "df = splink_datasets.fake_1000"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "d77b6eb8",
      "metadata": {},
      "source": [
        "## Load estimated model from previous tutorial\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "id": "619553a5",
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-06-22T08:08:06.115468Z",
          "iopub.status.busy": "2024-06-22T08:08:06.115158Z",
          "iopub.status.idle": "2024-06-22T08:08:06.295408Z",
          "shell.execute_reply": "2024-06-22T08:08:06.294871Z"
        },
        "tags": []
      },
      "outputs": [],
      "source": [
        "import json\n",
        "import urllib\n",
        "\n",
        "url = \"https://raw.githubusercontent.com/moj-analytical-services/splink/847e32508b1a9cdd7bcd2ca6c0a74e547fb69865/docs/demos/demo_settings/saved_model_from_demo.json\"\n",
        "\n",
        "with urllib.request.urlopen(url) as u:\n",
        "    settings = json.loads(u.read().decode())\n",
        "\n",
        "\n",
        "linker = Linker(df, settings, db_api=DuckDBAPI())"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "c1d97518",
      "metadata": {},
      "source": [
        "# Predicting match weights using the trained model\n",
        "\n",
        "We use `linker.inference.predict()` to run the model.\n",
        "\n",
        "Under the hood this will:\n",
        "\n",
        "- Generate all pairwise record comparisons that match at least one of the `blocking_rules_to_generate_predictions`\n",
        "\n",
        "- Use the rules specified in the `Comparisons` to evaluate the similarity of the input data\n",
        "\n",
        "- Use the estimated match weights, applying term frequency adjustments where requested to produce the final `match_weight` and `match_probability` scores\n",
        "\n",
        "Optionally, a `threshold_match_probability` or `threshold_match_weight` can be provided, which will drop any row where the predicted score is below the threshold.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "id": "ead23f3e",
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-06-22T08:08:06.298723Z",
          "iopub.status.busy": "2024-06-22T08:08:06.298474Z",
          "iopub.status.idle": "2024-06-22T08:08:06.707778Z",
          "shell.execute_reply": "2024-06-22T08:08:06.707043Z"
        },
        "tags": []
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n",
            " -- WARNING --\n",
            "You have called predict(), but there are some parameter estimates which have neither been estimated or specified in your settings dictionary.  To produce predictions the following untrained trained parameters will use default values.\n",
            "Comparison: 'email':\n",
            "    m values not fully trained\n"
          ]
        },
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>match_weight</th>\n",
              "      <th>match_probability</th>\n",
              "      <th>unique_id_l</th>\n",
              "      <th>unique_id_r</th>\n",
              "      <th>first_name_l</th>\n",
              "      <th>first_name_r</th>\n",
              "      <th>gamma_first_name</th>\n",
              "      <th>tf_first_name_l</th>\n",
              "      <th>tf_first_name_r</th>\n",
              "      <th>bf_first_name</th>\n",
              "      <th>bf_tf_adj_first_name</th>\n",
              "      <th>surname_l</th>\n",
              "      <th>surname_r</th>\n",
              "      <th>gamma_surname</th>\n",
              "      <th>tf_surname_l</th>\n",
              "      <th>tf_surname_r</th>\n",
              "      <th>bf_surname</th>\n",
              "      <th>bf_tf_adj_surname</th>\n",
              "      <th>dob_l</th>\n",
              "      <th>dob_r</th>\n",
              "      <th>gamma_dob</th>\n",
              "      <th>bf_dob</th>\n",
              "      <th>city_l</th>\n",
              "      <th>city_r</th>\n",
              "      <th>gamma_city</th>\n",
              "      <th>tf_city_l</th>\n",
              "      <th>tf_city_r</th>\n",
              "      <th>bf_city</th>\n",
              "      <th>bf_tf_adj_city</th>\n",
              "      <th>email_l</th>\n",
              "      <th>email_r</th>\n",
              "      <th>gamma_email</th>\n",
              "      <th>tf_email_l</th>\n",
              "      <th>tf_email_r</th>\n",
              "      <th>bf_email</th>\n",
              "      <th>bf_tf_adj_email</th>\n",
              "      <th>match_key</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>-1.749664</td>\n",
              "      <td>0.229211</td>\n",
              "      <td>324</td>\n",
              "      <td>326</td>\n",
              "      <td>Kai</td>\n",
              "      <td>Kai</td>\n",
              "      <td>4</td>\n",
              "      <td>0.006017</td>\n",
              "      <td>0.006017</td>\n",
              "      <td>84.821765</td>\n",
              "      <td>0.962892</td>\n",
              "      <td>None</td>\n",
              "      <td>Turner</td>\n",
              "      <td>-1</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0.007326</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>2018-12-31</td>\n",
              "      <td>2009-11-03</td>\n",
              "      <td>0</td>\n",
              "      <td>0.460743</td>\n",
              "      <td>London</td>\n",
              "      <td>London</td>\n",
              "      <td>1</td>\n",
              "      <td>0.212792</td>\n",
              "      <td>0.212792</td>\n",
              "      <td>10.20126</td>\n",
              "      <td>0.259162</td>\n",
              "      <td>k.t50eherand@z.ncom</td>\n",
              "      <td>None</td>\n",
              "      <td>-1</td>\n",
              "      <td>0.001267</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>-1.626076</td>\n",
              "      <td>0.244695</td>\n",
              "      <td>25</td>\n",
              "      <td>27</td>\n",
              "      <td>Gabriel</td>\n",
              "      <td>None</td>\n",
              "      <td>-1</td>\n",
              "      <td>0.001203</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>Thomas</td>\n",
              "      <td>Thomas</td>\n",
              "      <td>4</td>\n",
              "      <td>0.004884</td>\n",
              "      <td>0.004884</td>\n",
              "      <td>88.870507</td>\n",
              "      <td>1.001222</td>\n",
              "      <td>1977-09-13</td>\n",
              "      <td>1977-10-17</td>\n",
              "      <td>0</td>\n",
              "      <td>0.460743</td>\n",
              "      <td>London</td>\n",
              "      <td>London</td>\n",
              "      <td>1</td>\n",
              "      <td>0.212792</td>\n",
              "      <td>0.212792</td>\n",
              "      <td>10.20126</td>\n",
              "      <td>0.259162</td>\n",
              "      <td>gabriel.t54@nichols.info</td>\n",
              "      <td>None</td>\n",
              "      <td>-1</td>\n",
              "      <td>0.002535</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>-1.551265</td>\n",
              "      <td>0.254405</td>\n",
              "      <td>626</td>\n",
              "      <td>629</td>\n",
              "      <td>geeorGe</td>\n",
              "      <td>George</td>\n",
              "      <td>1</td>\n",
              "      <td>0.001203</td>\n",
              "      <td>0.014440</td>\n",
              "      <td>4.176727</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>Davidson</td>\n",
              "      <td>Davidson</td>\n",
              "      <td>4</td>\n",
              "      <td>0.007326</td>\n",
              "      <td>0.007326</td>\n",
              "      <td>88.870507</td>\n",
              "      <td>0.667482</td>\n",
              "      <td>1999-05-07</td>\n",
              "      <td>2000-05-06</td>\n",
              "      <td>0</td>\n",
              "      <td>0.460743</td>\n",
              "      <td>Southamptn</td>\n",
              "      <td>None</td>\n",
              "      <td>-1</td>\n",
              "      <td>0.001230</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1.00000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>None</td>\n",
              "      <td>gdavidson@johnson-brown.com</td>\n",
              "      <td>-1</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0.00507</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>-1.427735</td>\n",
              "      <td>0.270985</td>\n",
              "      <td>600</td>\n",
              "      <td>602</td>\n",
              "      <td>Toby</td>\n",
              "      <td>Toby</td>\n",
              "      <td>4</td>\n",
              "      <td>0.004813</td>\n",
              "      <td>0.004813</td>\n",
              "      <td>84.821765</td>\n",
              "      <td>1.203614</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "      <td>-1</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>2003-04-23</td>\n",
              "      <td>2013-03-21</td>\n",
              "      <td>0</td>\n",
              "      <td>0.460743</td>\n",
              "      <td>London</td>\n",
              "      <td>London</td>\n",
              "      <td>1</td>\n",
              "      <td>0.212792</td>\n",
              "      <td>0.212792</td>\n",
              "      <td>10.20126</td>\n",
              "      <td>0.259162</td>\n",
              "      <td>toby.d@menhez.com</td>\n",
              "      <td>None</td>\n",
              "      <td>-1</td>\n",
              "      <td>0.001267</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>-1.427735</td>\n",
              "      <td>0.270985</td>\n",
              "      <td>599</td>\n",
              "      <td>602</td>\n",
              "      <td>Toby</td>\n",
              "      <td>Toby</td>\n",
              "      <td>4</td>\n",
              "      <td>0.004813</td>\n",
              "      <td>0.004813</td>\n",
              "      <td>84.821765</td>\n",
              "      <td>1.203614</td>\n",
              "      <td>Haall</td>\n",
              "      <td>None</td>\n",
              "      <td>-1</td>\n",
              "      <td>0.001221</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>2003-04-23</td>\n",
              "      <td>2013-03-21</td>\n",
              "      <td>0</td>\n",
              "      <td>0.460743</td>\n",
              "      <td>London</td>\n",
              "      <td>London</td>\n",
              "      <td>1</td>\n",
              "      <td>0.212792</td>\n",
              "      <td>0.212792</td>\n",
              "      <td>10.20126</td>\n",
              "      <td>0.259162</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "      <td>-1</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   match_weight  match_probability  unique_id_l  unique_id_r first_name_l  \\\n",
              "0     -1.749664           0.229211          324          326          Kai   \n",
              "1     -1.626076           0.244695           25           27      Gabriel   \n",
              "2     -1.551265           0.254405          626          629      geeorGe   \n",
              "3     -1.427735           0.270985          600          602         Toby   \n",
              "4     -1.427735           0.270985          599          602         Toby   \n",
              "\n",
              "  first_name_r  gamma_first_name  tf_first_name_l  tf_first_name_r  \\\n",
              "0          Kai                 4         0.006017         0.006017   \n",
              "1         None                -1         0.001203              NaN   \n",
              "2       George                 1         0.001203         0.014440   \n",
              "3         Toby                 4         0.004813         0.004813   \n",
              "4         Toby                 4         0.004813         0.004813   \n",
              "\n",
              "   bf_first_name  bf_tf_adj_first_name surname_l surname_r  gamma_surname  \\\n",
              "0      84.821765              0.962892      None    Turner             -1   \n",
              "1       1.000000              1.000000    Thomas    Thomas              4   \n",
              "2       4.176727              1.000000  Davidson  Davidson              4   \n",
              "3      84.821765              1.203614      None      None             -1   \n",
              "4      84.821765              1.203614     Haall      None             -1   \n",
              "\n",
              "   tf_surname_l  tf_surname_r  bf_surname  bf_tf_adj_surname       dob_l  \\\n",
              "0           NaN      0.007326    1.000000           1.000000  2018-12-31   \n",
              "1      0.004884      0.004884   88.870507           1.001222  1977-09-13   \n",
              "2      0.007326      0.007326   88.870507           0.667482  1999-05-07   \n",
              "3           NaN           NaN    1.000000           1.000000  2003-04-23   \n",
              "4      0.001221           NaN    1.000000           1.000000  2003-04-23   \n",
              "\n",
              "        dob_r  gamma_dob    bf_dob      city_l  city_r  gamma_city  tf_city_l  \\\n",
              "0  2009-11-03          0  0.460743      London  London           1   0.212792   \n",
              "1  1977-10-17          0  0.460743      London  London           1   0.212792   \n",
              "2  2000-05-06          0  0.460743  Southamptn    None          -1   0.001230   \n",
              "3  2013-03-21          0  0.460743      London  London           1   0.212792   \n",
              "4  2013-03-21          0  0.460743      London  London           1   0.212792   \n",
              "\n",
              "   tf_city_r   bf_city  bf_tf_adj_city                   email_l  \\\n",
              "0   0.212792  10.20126        0.259162       k.t50eherand@z.ncom   \n",
              "1   0.212792  10.20126        0.259162  gabriel.t54@nichols.info   \n",
              "2        NaN   1.00000        1.000000                      None   \n",
              "3   0.212792  10.20126        0.259162         toby.d@menhez.com   \n",
              "4   0.212792  10.20126        0.259162                      None   \n",
              "\n",
              "                       email_r  gamma_email  tf_email_l  tf_email_r  bf_email  \\\n",
              "0                         None           -1    0.001267         NaN       1.0   \n",
              "1                         None           -1    0.002535         NaN       1.0   \n",
              "2  gdavidson@johnson-brown.com           -1         NaN     0.00507       1.0   \n",
              "3                         None           -1    0.001267         NaN       1.0   \n",
              "4                         None           -1         NaN         NaN       1.0   \n",
              "\n",
              "   bf_tf_adj_email match_key  \n",
              "0              1.0         0  \n",
              "1              1.0         1  \n",
              "2              1.0         1  \n",
              "3              1.0         0  \n",
              "4              1.0         0  "
            ]
          },
          "execution_count": 4,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "df_predictions = linker.inference.predict(threshold_match_probability=0.2)\n",
        "df_predictions.as_pandas_dataframe(limit=5)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "f00370bb",
      "metadata": {},
      "source": [
        "## Clustering\n",
        "\n",
        "The result of `linker.inference.predict()` is a list of pairwise record comparisons and their associated scores. For instance, if we have input records A, B, C and D, it could be represented conceptually as:\n",
        "\n",
        "```\n",
        "A -> B with score 0.9\n",
        "B -> C with score 0.95\n",
        "C -> D with score 0.1\n",
        "D -> E with score 0.99\n",
        "```\n",
        "\n",
        "Often, an alternative representation of this result is more useful, where each row is an input record, and where records link, they are assigned to the same cluster.\n",
        "\n",
        "With a score threshold of 0.5, the above data could be represented conceptually as:\n",
        "\n",
        "```\n",
        "ID, Cluster ID\n",
        "A,  1\n",
        "B,  1\n",
        "C,  1\n",
        "D,  2\n",
        "E,  2\n",
        "```\n",
        "\n",
        "The algorithm that converts between the pairwise results and the clusters is called connected components, and it is included in Splink. You can use it as follows:\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "id": "257ae717",
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-06-22T08:08:06.711722Z",
          "iopub.status.busy": "2024-06-22T08:08:06.711425Z",
          "iopub.status.idle": "2024-06-22T08:08:06.756664Z",
          "shell.execute_reply": "2024-06-22T08:08:06.755985Z"
        }
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Completed iteration 1, root rows count 2\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Completed iteration 2, root rows count 0\n"
          ]
        },
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>cluster_id</th>\n",
              "      <th>unique_id</th>\n",
              "      <th>first_name</th>\n",
              "      <th>surname</th>\n",
              "      <th>dob</th>\n",
              "      <th>city</th>\n",
              "      <th>email</th>\n",
              "      <th>cluster</th>\n",
              "      <th>__splink_salt</th>\n",
              "      <th>tf_surname</th>\n",
              "      <th>tf_email</th>\n",
              "      <th>tf_city</th>\n",
              "      <th>tf_first_name</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>Robert</td>\n",
              "      <td>Alan</td>\n",
              "      <td>1971-06-24</td>\n",
              "      <td>None</td>\n",
              "      <td>robert255@smith.net</td>\n",
              "      <td>0</td>\n",
              "      <td>0.012924</td>\n",
              "      <td>0.001221</td>\n",
              "      <td>0.001267</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0.003610</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>Robert</td>\n",
              "      <td>Allen</td>\n",
              "      <td>1971-05-24</td>\n",
              "      <td>None</td>\n",
              "      <td>roberta25@smith.net</td>\n",
              "      <td>0</td>\n",
              "      <td>0.478756</td>\n",
              "      <td>0.002442</td>\n",
              "      <td>0.002535</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0.003610</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>Rob</td>\n",
              "      <td>Allen</td>\n",
              "      <td>1971-06-24</td>\n",
              "      <td>London</td>\n",
              "      <td>roberta25@smith.net</td>\n",
              "      <td>0</td>\n",
              "      <td>0.409662</td>\n",
              "      <td>0.002442</td>\n",
              "      <td>0.002535</td>\n",
              "      <td>0.212792</td>\n",
              "      <td>0.001203</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>3</td>\n",
              "      <td>3</td>\n",
              "      <td>Robert</td>\n",
              "      <td>Alen</td>\n",
              "      <td>1971-06-24</td>\n",
              "      <td>Lonon</td>\n",
              "      <td>None</td>\n",
              "      <td>0</td>\n",
              "      <td>0.311029</td>\n",
              "      <td>0.001221</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0.007380</td>\n",
              "      <td>0.003610</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>4</td>\n",
              "      <td>4</td>\n",
              "      <td>Grace</td>\n",
              "      <td>None</td>\n",
              "      <td>1997-04-26</td>\n",
              "      <td>Hull</td>\n",
              "      <td>grace.kelly52@jones.com</td>\n",
              "      <td>1</td>\n",
              "      <td>0.486141</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0.002535</td>\n",
              "      <td>0.001230</td>\n",
              "      <td>0.006017</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>5</td>\n",
              "      <td>5</td>\n",
              "      <td>Grace</td>\n",
              "      <td>Kelly</td>\n",
              "      <td>1991-04-26</td>\n",
              "      <td>None</td>\n",
              "      <td>grace.kelly52@jones.com</td>\n",
              "      <td>1</td>\n",
              "      <td>0.434566</td>\n",
              "      <td>0.002442</td>\n",
              "      <td>0.002535</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0.006017</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>6</td>\n",
              "      <td>6</td>\n",
              "      <td>Logan</td>\n",
              "      <td>pMurphy</td>\n",
              "      <td>1973-08-01</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "      <td>2</td>\n",
              "      <td>0.423760</td>\n",
              "      <td>0.001221</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0.012034</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>7</td>\n",
              "      <td>7</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "      <td>2015-03-03</td>\n",
              "      <td>Portsmouth</td>\n",
              "      <td>evied56@harris-bailey.net</td>\n",
              "      <td>3</td>\n",
              "      <td>0.683689</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0.002535</td>\n",
              "      <td>0.017220</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>8</td>\n",
              "      <td>8</td>\n",
              "      <td>None</td>\n",
              "      <td>Dean</td>\n",
              "      <td>2015-03-03</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "      <td>3</td>\n",
              "      <td>0.553086</td>\n",
              "      <td>0.003663</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>8</td>\n",
              "      <td>9</td>\n",
              "      <td>Evie</td>\n",
              "      <td>Dean</td>\n",
              "      <td>2015-03-03</td>\n",
              "      <td>Pootsmruth</td>\n",
              "      <td>evihd56@earris-bailey.net</td>\n",
              "      <td>3</td>\n",
              "      <td>0.753070</td>\n",
              "      <td>0.003663</td>\n",
              "      <td>0.001267</td>\n",
              "      <td>0.001230</td>\n",
              "      <td>0.008424</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   cluster_id  unique_id first_name  surname         dob        city  \\\n",
              "0           0          0     Robert     Alan  1971-06-24        None   \n",
              "1           1          1     Robert    Allen  1971-05-24        None   \n",
              "2           1          2        Rob    Allen  1971-06-24      London   \n",
              "3           3          3     Robert     Alen  1971-06-24       Lonon   \n",
              "4           4          4      Grace     None  1997-04-26        Hull   \n",
              "5           5          5      Grace    Kelly  1991-04-26        None   \n",
              "6           6          6      Logan  pMurphy  1973-08-01        None   \n",
              "7           7          7       None     None  2015-03-03  Portsmouth   \n",
              "8           8          8       None     Dean  2015-03-03        None   \n",
              "9           8          9       Evie     Dean  2015-03-03  Pootsmruth   \n",
              "\n",
              "                       email  cluster  __splink_salt  tf_surname  tf_email  \\\n",
              "0        robert255@smith.net        0       0.012924    0.001221  0.001267   \n",
              "1        roberta25@smith.net        0       0.478756    0.002442  0.002535   \n",
              "2        roberta25@smith.net        0       0.409662    0.002442  0.002535   \n",
              "3                       None        0       0.311029    0.001221       NaN   \n",
              "4    grace.kelly52@jones.com        1       0.486141         NaN  0.002535   \n",
              "5    grace.kelly52@jones.com        1       0.434566    0.002442  0.002535   \n",
              "6                       None        2       0.423760    0.001221       NaN   \n",
              "7  evied56@harris-bailey.net        3       0.683689         NaN  0.002535   \n",
              "8                       None        3       0.553086    0.003663       NaN   \n",
              "9  evihd56@earris-bailey.net        3       0.753070    0.003663  0.001267   \n",
              "\n",
              "    tf_city  tf_first_name  \n",
              "0       NaN       0.003610  \n",
              "1       NaN       0.003610  \n",
              "2  0.212792       0.001203  \n",
              "3  0.007380       0.003610  \n",
              "4  0.001230       0.006017  \n",
              "5       NaN       0.006017  \n",
              "6       NaN       0.012034  \n",
              "7  0.017220            NaN  \n",
              "8       NaN            NaN  \n",
              "9  0.001230       0.008424  "
            ]
          },
          "execution_count": 5,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "clusters = linker.clustering.cluster_pairwise_predictions_at_threshold(\n",
        "    df_predictions, threshold_match_probability=0.5\n",
        ")\n",
        "clusters.as_pandas_dataframe(limit=10)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "id": "b973f53f-6d57-4c79-a87d-fbad40f303f1",
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-06-22T08:08:06.760329Z",
          "iopub.status.busy": "2024-06-22T08:08:06.760043Z",
          "iopub.status.idle": "2024-06-22T08:08:06.788279Z",
          "shell.execute_reply": "2024-06-22T08:08:06.787675Z"
        }
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>match_weight</th>\n",
              "      <th>match_probability</th>\n",
              "      <th>unique_id_l</th>\n",
              "      <th>unique_id_r</th>\n",
              "      <th>first_name_l</th>\n",
              "      <th>first_name_r</th>\n",
              "      <th>gamma_first_name</th>\n",
              "      <th>tf_first_name_l</th>\n",
              "      <th>tf_first_name_r</th>\n",
              "      <th>bf_first_name</th>\n",
              "      <th>bf_tf_adj_first_name</th>\n",
              "      <th>surname_l</th>\n",
              "      <th>surname_r</th>\n",
              "      <th>gamma_surname</th>\n",
              "      <th>tf_surname_l</th>\n",
              "      <th>tf_surname_r</th>\n",
              "      <th>bf_surname</th>\n",
              "      <th>bf_tf_adj_surname</th>\n",
              "      <th>dob_l</th>\n",
              "      <th>dob_r</th>\n",
              "      <th>gamma_dob</th>\n",
              "      <th>bf_dob</th>\n",
              "      <th>city_l</th>\n",
              "      <th>city_r</th>\n",
              "      <th>gamma_city</th>\n",
              "      <th>tf_city_l</th>\n",
              "      <th>tf_city_r</th>\n",
              "      <th>bf_city</th>\n",
              "      <th>bf_tf_adj_city</th>\n",
              "      <th>email_l</th>\n",
              "      <th>email_r</th>\n",
              "      <th>gamma_email</th>\n",
              "      <th>tf_email_l</th>\n",
              "      <th>tf_email_r</th>\n",
              "      <th>bf_email</th>\n",
              "      <th>bf_tf_adj_email</th>\n",
              "      <th>match_key</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>-1.749664</td>\n",
              "      <td>0.229211</td>\n",
              "      <td>324</td>\n",
              "      <td>326</td>\n",
              "      <td>Kai</td>\n",
              "      <td>Kai</td>\n",
              "      <td>4</td>\n",
              "      <td>0.006017</td>\n",
              "      <td>0.006017</td>\n",
              "      <td>84.821765</td>\n",
              "      <td>0.962892</td>\n",
              "      <td>None</td>\n",
              "      <td>Turner</td>\n",
              "      <td>-1</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0.007326</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>2018-12-31</td>\n",
              "      <td>2009-11-03</td>\n",
              "      <td>0</td>\n",
              "      <td>0.460743</td>\n",
              "      <td>London</td>\n",
              "      <td>London</td>\n",
              "      <td>1</td>\n",
              "      <td>0.212792</td>\n",
              "      <td>0.212792</td>\n",
              "      <td>10.20126</td>\n",
              "      <td>0.259162</td>\n",
              "      <td>k.t50eherand@z.ncom</td>\n",
              "      <td>None</td>\n",
              "      <td>-1</td>\n",
              "      <td>0.001267</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>-1.626076</td>\n",
              "      <td>0.244695</td>\n",
              "      <td>25</td>\n",
              "      <td>27</td>\n",
              "      <td>Gabriel</td>\n",
              "      <td>None</td>\n",
              "      <td>-1</td>\n",
              "      <td>0.001203</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>Thomas</td>\n",
              "      <td>Thomas</td>\n",
              "      <td>4</td>\n",
              "      <td>0.004884</td>\n",
              "      <td>0.004884</td>\n",
              "      <td>88.870507</td>\n",
              "      <td>1.001222</td>\n",
              "      <td>1977-09-13</td>\n",
              "      <td>1977-10-17</td>\n",
              "      <td>0</td>\n",
              "      <td>0.460743</td>\n",
              "      <td>London</td>\n",
              "      <td>London</td>\n",
              "      <td>1</td>\n",
              "      <td>0.212792</td>\n",
              "      <td>0.212792</td>\n",
              "      <td>10.20126</td>\n",
              "      <td>0.259162</td>\n",
              "      <td>gabriel.t54@nichols.info</td>\n",
              "      <td>None</td>\n",
              "      <td>-1</td>\n",
              "      <td>0.002535</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   match_weight  match_probability  unique_id_l  unique_id_r first_name_l  \\\n",
              "0     -1.749664           0.229211          324          326          Kai   \n",
              "1     -1.626076           0.244695           25           27      Gabriel   \n",
              "\n",
              "  first_name_r  gamma_first_name  tf_first_name_l  tf_first_name_r  \\\n",
              "0          Kai                 4         0.006017         0.006017   \n",
              "1         None                -1         0.001203              NaN   \n",
              "\n",
              "   bf_first_name  bf_tf_adj_first_name surname_l surname_r  gamma_surname  \\\n",
              "0      84.821765              0.962892      None    Turner             -1   \n",
              "1       1.000000              1.000000    Thomas    Thomas              4   \n",
              "\n",
              "   tf_surname_l  tf_surname_r  bf_surname  bf_tf_adj_surname       dob_l  \\\n",
              "0           NaN      0.007326    1.000000           1.000000  2018-12-31   \n",
              "1      0.004884      0.004884   88.870507           1.001222  1977-09-13   \n",
              "\n",
              "        dob_r  gamma_dob    bf_dob  city_l  city_r  gamma_city  tf_city_l  \\\n",
              "0  2009-11-03          0  0.460743  London  London           1   0.212792   \n",
              "1  1977-10-17          0  0.460743  London  London           1   0.212792   \n",
              "\n",
              "   tf_city_r   bf_city  bf_tf_adj_city                   email_l email_r  \\\n",
              "0   0.212792  10.20126        0.259162       k.t50eherand@z.ncom    None   \n",
              "1   0.212792  10.20126        0.259162  gabriel.t54@nichols.info    None   \n",
              "\n",
              "   gamma_email  tf_email_l  tf_email_r  bf_email  bf_tf_adj_email match_key  \n",
              "0           -1    0.001267         NaN       1.0              1.0         0  \n",
              "1           -1    0.002535         NaN       1.0              1.0         1  "
            ]
          },
          "execution_count": 6,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "sql = f\"\"\"\n",
        "select *\n",
        "from {df_predictions.physical_name}\n",
        "limit 2\n",
        "\"\"\"\n",
        "linker.misc.query_sql(sql)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "177c5013",
      "metadata": {},
      "source": [
        "!!! note \"Further Reading\"\n",
        ":material-tools: For more on the prediction tools in Splink, please refer to the [Prediction API documentation](../../api_docs/inference.md).\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "b7cae5d7",
      "metadata": {},
      "source": [
        "## Next steps\n",
        "\n",
        "Now we have made predictions with a model, we can move on to visualising it to understand how it is working.\n"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.10.8"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}
